Synchronizing holographic displays and 3d objects with physical video panels

ABSTRACT

A method of coordinating a mixed-reality (MR) configured head-mounted display (HMD) with a separate media device to enable a synchronized user experience. The method includes establishing a communication channel between the HMD and the media device. At least one of the following is performed via the communication channel: accessing content on the media device or executing control commands on the media device based on an interface displayed by the HMD, or detecting media content presented by the media device and synchronizing display of MR content on the HMD and the detected media content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of U.S. Provisional PatentApplication Ser. No. 62/539,808, filed Aug. 1, 2017, which isincorporated herein by reference in its entirety.

COPYRIGHT DISCLAIMER

A portion of the disclosure of this patent document may contain materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the patent and trademarkoffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

One or more embodiments relate generally to holographic content, and inparticular, to synchronizing holographic content with display and userinterface (UI) technology.

BACKGROUND

Virtual reality devices, such as head-mounted devices, may be used in avariety of real and/or virtual world environments and contexts.Augmented reality devices are types of virtual reality devices that cansupport direct or indirect views of a real world environment along withaugmented reality objects digitally projected on the real world scene.Augmented reality devices can also operate as scene-aware devices thathave an understanding of a real world environment defined as anaugmented reality environment (i.e., virtual environment) supported bythe augmented reality device. An augmented reality device can supportpresentation of the augmented reality objects, which are virtualizedentities (e.g., holographic content or mixed-reality content), that arerendered for a user associated with the augmented reality device.

SUMMARY

One or more embodiments relate to synchronizing holographic content withdisplay and user interface (UI) technology. In some embodiments, amethod of coordinating a mixed-reality (MR) configured head-mounteddisplay (HMD) with a separate media device to enable a synchronized userexperience. The method includes establishing a communication channelbetween the HMD and the media device. At least one of the following isperformed via the communication channel: accessing content on the mediadevice or executing control commands on the media device based on aninterface displayed by the HMD, or detecting media content presented bythe media device and synchronizing display of MR content on the HMD anddetected media content.

In some embodiments, an apparatus comprises a memory storinginstructions. At least one processor executes the instructions includinga process configured to: establish a communication channel between an MRHMD and a media device; and perform using the communication channel, atleast one of: accessing content on the media device or executing controlcommands on the media device based on an interface displayed by the HMD;or detecting media content presented by the media device andsynchronizing display of MR content on the HMD and the detected mediacontent.

In some embodiments, a non-transitory processor-readable medium thatincludes a program that when executed by a processor performs a methodcomprising establishing a communication channel between an MR HMD and amedia device, and performing using the communication channel, at leastone of: accessing content on the media device or executing controlcommands on the media device based on an interface displayed by the HMD;or detecting media content presented by the media device andsynchronizing display of MR content on the HMD and detected mediacontent.

These and other features, aspects and advantages of the one or moreembodiments will become understood with reference to the followingdescription, appended claims and accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an electronic system with an overlay enhancement mechanism,according to some embodiments;

FIG. 2 shows an example high-level block diagram of a system, accordingto some embodiments;

FIG. 3 shows an example television (TV) device and holographic displaydevice discovery and pairing, according to some embodiments;

FIG. 4 shows a mixed reality (MR) framework pairing flow diagram,according to some embodiments;

FIG. 5 shows an example holograph app user interface (UI), according tosome embodiments;

FIG. 6 shows a virtual TV UI panel, according to some embodiments;

FIG. 7 shows a virtual preview panel, according to some embodiments;

FIG. 8 shows an example movie event holographic display, according tosome embodiments;

FIG. 9 shows a block diagram for holographic object insertionprocessing, according to some embodiments;

FIG. 10 shows an example of dynamic advertising insertion in a virtualworld environment, according to some embodiments;

FIG. 11 shows an example of a smart TV and UI, according to someembodiments;

FIG. 12 shows a block diagram of an example system layout, according tosome embodiments;

FIG. 13 shows an example diagram showing the sharing of a viewingexperience by synchronizing media devices and projecting avatars inrespective space, according to some embodiments;

FIG. 14 shows use of simultaneous visual input and network communicationfor providing precise determination of distance and position in 3Dspace, according to some embodiments;

FIG. 15 shows a block diagram for synchronizing holographic content withdisplay and UI technology processing, according to some embodiments; and

FIG. 16 is an exemplary high-level block diagram showing an informationprocessing system comprising a computer system useful for implementingdisclosed embodiments.

DETAILED DESCRIPTION

The following description is made for the purpose of illustrating thegeneral principles of one or more embodiments and is not meant to limitthe inventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc.

Some embodiments provide synchronizing of holographic content withdisplay and user interface (UI) technology. In some embodiments, amethod of coordinating a mixed-reality (MR) configured head-mounteddisplay (HMD) with a separate media device to enable a synchronized userexperience. The method includes establishing a communication channelbetween the HMD and the media device. At least one of the following isperformed via the communication channel: accessing content on the mediadevice or executing control commands on the media device based on aninterface displayed by the HMD, and detecting media presented by themedia device and synchronizing display of content associated with thedetected media on the HMD.

The term “image” referred to herein may include a two-dimensional image,three-dimensional image, video frame, a computer file representation, animage from a camera, a video frame, or a combination thereof. Forexample, the image may be a machine readable digital file, a physicalphotograph, a digital photograph, a motion picture frame, a video frame,an x-ray image, a scanned image, or a combination thereof. The image maybe generated from pixels arranged in a rectangular array. The image mayinclude an x-axis along the direction of the rows and a y-axis along thedirection of the columns. The term “image” may indicate a still image ora moving picture of a video, i.e., the latter indicating the videoitself. The term “image” may include a partial or the entire screenimage displayable on a display besides a still image (e.g., aphotograph) or a video. In addition, the term “image” may alsooriginally include a displayable screen image itself such as a userinterface or a webpage besides a still image (e.g., a photograph) or avideo.

In some embodiments, holographic technology that is able to project 3Drepresentations as if they were in the same space as the user isgenerated. The holographic technology may include a head-mounted-display(HMD) (e.g., HMD 320, FIG. 3) that is worn by a user. The HMD showsholograms by projecting light onto a transparent screen that is wornover the user's eyes. In some embodiments, the holographic technology iscombined with display technology (such as a TV device 250, FIG. 2, a TVdevice 310, FIG. 3, etc.), including Large Format Display/LFD),synchronizing the HMD and TV and providing for the two to work togetherto heighten the user experience.

In some embodiments, users are able to view and interact with hologramspositioned relative to the screen that they are watching content on.Users can simultaneously enjoy an uncompromised, high-resolutionexperience on the screen, while also viewing holograms off-screen.Holograms displayed to the user supplement and synchronize with thecontent displayed on the screen being viewed, such as movies, TV shows,shopping experiences, gaming, advertisements, etc.

FIG. 1 shows an electronic system 50 with an overlay enhancementmechanism in some embodiments. The electronic system 50 includes a firstdevice 52, such as a client or a server, connected to a second device56, such as a client or server. The first device 52 may communicate withthe second device 56 with a communication path 54, such as a wireless orwired network.

In one example, the first device 52 may be of any of a variety ofdisplay devices, such as ultra-high definition (UD), 4K (8K, etc.)display devices, such as a UD television (UDTV), 4K TV, 8K TV, tabletdevice, smart phone, personal digital assistant (PDA), a notebookcomputer, a liquid crystal display (LCD) system, a wearable device,mobile computing device, projection device, or other multi-functionaldisplays or entertainment devices. The first device 52 may coupledirectly or indirectly to the communication path 54 to communicate withthe second device 56 or may be a stand-alone device.

For illustrative purposes, the display system 50 is described with thefirst device 52 as a display device, although it is understood that thefirst device 52 may be a variety of different types of devices. Forexample, the first device 52 may also be a device for presenting imagesor a multi-media presentation. A multi-media presentation may be apresentation including sound, a sequence of streaming images or a videofeed, or a combination thereof. As an example, the first device 52 maybe a UDTV, or any other type of UD display device (e.g., monitor, videopanel, HUD, smart telephone, tablet device, video device, gaming device,etc.).

The second device 56 may be any of a variety of centralized ordecentralized computing devices, image or video transmission devices.For example, the second device 56 may be a multimedia computer, atablet, a laptop computer, a desktop computer, a video game console,grid-computing resources, a virtualized computer resource, cloudcomputing resource, routers, switches, peer-to-peer distributedcomputing devices, a media playback device, a Digital Video Disk (DVD)player, a three-dimension enabled DVD player, BDP, a recording device,such as a camera or video camera, or a combination thereof. In anotherexample, the second device 56 may be a signal receiver for receivingbroadcast or live stream signals, such as a television receiver, STB, acable box, a satellite dish receiver, or a web enabled device.

The second device 56 may be centralized in a single room, distributedacross different rooms, distributed across different geographicallocations, embedded within a telecommunications network, etc. The seconddevice 56 may have a means for coupling with the communication path 54to communicate with the first device 52.

For illustrative purposes, the electronic system 50 is described withthe second device 56 as a computing device, although it is understoodthat the second device 56 may be different types of devices. Also, forillustrative purposes, the display system 50 is shown with the seconddevice 56 and the first device 52 as end points of the communicationpath 54, although it is understood that the display system 50 may have adifferent partition between the first device 52, the second device 56,and the communication path 54. For example, the first device 52, thesecond device 56, or a combination thereof may also function as part ofthe communication path 54.

The communication path 54 may be a variety of networks. For example, thecommunication path 54 may include wireless communication, wiredcommunication, optical, ultrasonic, or the combination thereof.Satellite communication, cellular communication, BLUETOOTH®, InfraredData Association standard (IrDA), wireless fidelity (WiFi), andworldwide interoperability for microwave access (WiMAX) are examples ofwireless communication that may be included in the communication path54. Ethernet, digital subscriber line (DSL), fiber to the home (FTTH),high-definition multimedia interface (HDMI) cable, and plain oldtelephone service (POTS) are examples of wired communication that may beincluded in the communication path 54.

Further, the communication path 54 may traverse a number of networktopologies and distances. For example, the communication path 54 mayinclude direct connection, personal area network (PAN), local areanetwork (LAN), metropolitan area network (MAN), wide area network (WAN),or a combination thereof.

FIG. 2 shows a high-level block diagram of a system 200, according tosome embodiments. In some embodiments, the system 200 may process inputvideo images from an input source 210 (or device 52 or 56, FIG. 1)received at an input node 201 (e.g., from the communication path 54,FIG. 1) using an overlay enhancement processor 700 (e.g., an integratedcircuit (IC), hardware circuitry, a multi-core processor, an applicationspecific IC (ASIC), CPU, hybrid device and application programminginterface (API), etc.), output video images at the output node 240(e.g., to the communication path 54) and display the images on an outputsource 250 (or device 52, FIG. 1). In some embodiments, the display foran output source 250 may be a physical device for presenting the imageor multi-media presentations. For example, the display may be a screen,including a liquid crystal display (LCD) panel, a plasma screen, aprojection screen, a heads-up-display (HUD), etc. In other embodiments,the display may be projected on an object or reflection device.

In some embodiments, the input video images may be provided from aninput source 210, which may be transmitted/received wirelessly orthrough a wired interface (e.g., the communication path 54, FIG. 1) andmay include uncompressed/compressed video content. In some embodiments,wireline or wireless communication of video imaging content in system200 may include communication on/over one or more of a Ethernet,telephone (e.g., POTS), cable, power-line, and fiber optic systems,and/or a wireless system comprising one or more of a code divisionmultiple access (CDMA or CDMA2000) communication system, a frequencydivision multiple access (FDMA) system, a time division multiple access(TDMA) system such as GSM/GPRS (General Packet Radio Service)/EDGE(enhanced data GSM environment), a TETRA (Terrestrial Trunked Radio)mobile telephone system, a wideband code division multiple access(WCDMA) system, a high data rate (1× enhanced voice-data only (EV-DO) or1×EV-DO Gold Multicast) system, an IEEE 802.11x system, a DMB (DigitalMultimedia Broadcasting) system, an orthogonal frequency divisionmultiple access (OFDM) system, a DVB-H (Digital VideoBroadcasting-Handheld) system, etc.

In some embodiments, the video input source 210 may be transmitted overa wireless network (e.g., Internet, local area network (LAN, wide-areanetwork (WAN), personal area network (PAN), campus wireless network(CAN), metropolitan area network (MAN), etc., e.g., the communicationpath 54, FIG. 1). The input source 210 may arise from a cloud-basedsystem, a server, a broadcast station, a video device/player, a videocamera, a mobile device, etc.

In some embodiments, the video image input from the input source may bedecoded/encoded (e.g., via a decoder/encoder) prior to arriving at theinput node 201. The video image output from the output node 240 to anoutput source 250 may be encoded/decoded prior to arriving at the outputnode 240. In some embodiments, the output source 250 may receive theoutput image from the output node 240 wirelessly or through a wireinterface.

In some embodiments, compressed video image content from an input source210 may provide analog or digital video from a broadcast, computernetwork, a DVD or other computer readable storage medium, or any othersuitable source of video signals. In one embodiment, the compressedvideo from an input source 210 may be coupled to a receiver via a radiofrequency interface (such as ATSC broadcasts), a computer network, or awired or wireless connection such as a component video cable, a DVI orHDMI video interface, etc. In one embodiment, an uncompressed videoimage from an input source 210 may include one or more of a videocamera, or a memory device such as a disk drive coupled to a receiver byany suitable video interface. The uncompressed video from an inputsource 210 may provide uncompressed video in analog or digital formincluding in the form of luminance and chrominance, in individual colorintensities such as red, green and blue, etc., or in any other suitableformat.

In some embodiments, digital video formats for input video content froman input source 210 may include different frame rates, different numbersof lines and rows of pixels, interlaced and non-interlaced, etc. Forexample, movies are typically produced at 24 fps (frames per second)format, NTSC is 30 frames per second and PAL is 25 frames per second.The format may be interlaced or progressive. For example, highdefinition formats (such as supported by digital video broadcaststandards) may be modified to a format that fits the format of a displaydevice of an output source 250. The display device of an output source250 may be configured to receive NTSC, PAL, ATSC, DVB/T, etc.), ordisplay device running at frame rates, such as 70 Hz, 75 Hz, 80 Hz, etc.

In some embodiments, a processor (e.g., image processing processor 700,FIG. 2, processor 1601, FIG. 16, etc.) may perform the following. Onefeature of some embodiments includes a screen-oriented holographic userinterface (UI) experience. A holographic UI is shown that is positionedrelative to the display (TV, LFD, etc.). When the user wears the HMD,they see a UI that appears to be floating outside of the display. Theholographic UI provides users the ability to interact by using gestures,taps, or voice commands. Holograms may be projected relative to thedisplay, the user, or other objects detected in the environment space(e.g. a coffee table, etc.). The UI may be scaled according to the sizeof the display, providing for the effect of the UI “surrounding” thedisplay.

In some embodiments, interactions with the holograms may affect thedisplay itself. For example, selecting a holographic menu item maylaunch an application on the display. Users may use the entirety of theenvironmental space they are inside (e.g., a room, etc.) to view menus,and retain the ability to watch video content on the higher-resolutiondisplay. In some embodiments, the UI may include holograms that arepresented in the form of “tiles” or rectangles that can be selected inorder to perform an action. Holograms such as tiles may be generatedfrom data that the display sends. For example, the display may sendinformation about its size, allowing holograms to “surround” thedisplay, or it may send information about apps or content available onthe display, providing the HMD to generate holograms that allow the userto open those apps or play the content. Holograms may be interacted withvia actions such as gazing (short and long gaze), tapping (using fingergesture), voice commands, etc.

In some embodiments, detecting the position of the display is performedby using augmented reality (AR) Tracking. In order for a holographic UIto be positioned relative the display, the HMD must first know thedisplays' position. In some embodiments, determining the displays'position may be accomplished using AR tracking. The display may show apattern on its screen that allows the HMD to recognize it using computervision, and determine its position in world space. In addition todetermining the position of the display, the HMD also receivesinformation necessary to determine the display's dimensions.

In some embodiments, content-aware holographic asset insertion forentertainment (e.g. content video, games, etc.) is provided. The displayand HMD communicate with each other to synchronize the display of 3Dholographic assets with video and other content. Examples may includecontent video. For example, a movie begins playing on the display. Atcertain time indexes in the movie, the display instructs the HMD todisplay holograms related to the scenes of the movie, increasing userimmersion. Another example is gaming where a game is being played on thedisplay. The display instructs the HMD to play effects relevant to whatis occurring in the game.

In some embodiments, dynamic advertisement insertion (DAI) using a 3Dholographic asset server or store (e.g., 3D asset store 1250, FIG. 12)for holograms in the HMD is provided. In traditional video applications,a targeted advertisement can be served and shown to a user dynamicallybased on the user's profiling data. These advertisements are typicallyin 2D video formats. In some embodiments, the user experience withadvertisements is enhanced using 3D holographic assets. Given a user'sprofiling data on both the display and HMD, the display and HMDcoordinate to retrieve and render an advertisement in the form of aholographic asset into the user's HMD virtual environment. For example,dynamically inserting 3D holographic ad objects (e.g. RED BULL® airplaneflying around in your HMD display while a RED BULL® advertisement isshown on a TV screen (see, e.g., FIG. 10).

In some embodiments, the display and HMD dynamically pull holographicassets from an asset pool/store. The DAI is available via linear, live,or video on demand (VOD) content. 3D holographic product placement isprovided, which allows an online purchase (e.g., a holographic RED BULL®drink placed on top of a coffee table in front of a TV, FIG. 10). Theuser can interact with the RED BULL® drink to purchase via online fordelivery to your house). This user interaction provides the user animmersive experience when viewing the holographic asset with the intentof making the advertisement more effective. In one example, contentvideo is playing on a visual display. At a given time, the visualdisplay will instruct the HMD to insert dynamic advertisements in theform of holograms related to the content video. The holograms are thensynchronized with video on the display.

In some embodiments, holographic asset insertion is provided forE-commerce shopping experiences. In traditional shopping experiences,sellable products, experiences, services, etc., are shown to consumersin the form of text, images, and videos. The shopping experience may beenhanced using holographic assets, holograms allowing users toexperience a product, experience, service, etc. in a virtual space witha more accurate impression. Users in the virtual environment of the HMDare able to walk around in a real physical space to visualize and/orcontrol a 3D virtualized representation of a product, experience,service, etc. The intent is to provide the user with a betterunderstanding before making a purchase. The display and HMD pullholographic assets from an asset pool/store. Payment may be made in thedisplay or HMD device by means of e-commerce (e.g., Samsung Pay). Duringa user's interactions with the display, products may show up on thescreen or in a video. The display communicates with the HMD aboutproducts being shown on the screen, and the HMD projects hologramsrelating to what is shown on the display, allowing the user to interactwith and purchase products. In one example, a user interacts with thedisplay and uses the HMD device. A holographic asset is inserted in theuser's virtual environment with the option to be purchased. The user isable to visualize, walk around, and interact with the scaled product,experience, service, etc. A user may purchase the product, experience,service, etc. Examples of items that may be purchased include, but arenot limited to, general products, consumer electronics, appliances,music, food, vehicles, fashion, furniture, travel destinations, localservices, etc.

In some embodiments, a social TV platform is provided that uses virtualavatars in a holographic environment. With the Social TV Platform, usersare able to watch video content and communicate with each other, withoutbeing in the same real physical space. The purpose is to enhance thesocial experience surrounding today's TV display. Each user will be ableto join a “virtual room” to view the same video content. The same videocontent is synchronously displayed on each user's display device. Usersin a given “virtual room” are able to control the video content. Eachuser may be represented as a “virtual avatar.” A virtual avatar is avirtual representation of the user. Users in a “virtual room” are ableto uniquely identify each other by the design of their virtual avatar.Given the sensors on the HMD device (e.g., accelerometer, gyroscope,compass, etc.), the virtual avatar is capable of mimicking the headmovement of the user, to provide a realistic representation of thedirection in which a user is gazing. Users in a “virtual room” are ableto communicate with each other via voice, which may optionally trigger amouth animation on the virtual avatar. Users in the social TV platformare able to add/remove each other via a “friend list.” Virtual avatarsmay be positioned relative to the display device (e.g., the TV, etc.).

In some embodiments, a mixed reality ecosystem for a holographic app andasset store is provided. The MR ecosystem may be composed of an HMD(e.g., HMD 320, FIG. 3, HMD device 1220, FIG. 12), a TV or media device(e.g., TV device 250, FIG. 2, TV device 310, FIG. 3, media device 1210,FIG. 12), a holographic app store, and a holographic 3D asset-store(e.g., 3D asset store 1250, FIG. 12). In some embodiments, there areseveral entities that interact with the ecosystem such as applicationsoftware developers for TV and HMD platforms, 3D holographic assetdevelopers, publishers (for DAI scenarios), and TV viewers. An appdeveloper builds TV web application for TV and media device sand submitsthe TV web application to a TV app store. An app developer builds UIapplications for the HMD and submits this companion app to theholographic app store. The holographic apps may be downloaded from theholographic app store into the HMD device. The holographic apps have aholographic client software library as part of the application. ATV-side holographic client software app or module exists as part of theTV firmware. A holographic 3D asset server may be a CDN Http serverwhich contains various (unity) asset bundles for content video developedby 3D holographic asset developers around the world. A Publisher maydecide when to dynamically insert/render 3D holographic objects duringcontent playback.

In some embodiments, a business model for the holographic app and assetstore creates a holographic 3D asset-store similarly as a popular musicapp store would, to allow app developers to purchase holographic objectswith imaginative effects to enhance the viewing experience for theirapps. The new asset-store system provides for any 3D holographic assetdeveloper to create, submit, and sell their assets for holographic apps.

FIG. 3 shows an example TV device 310 and HMD 320 discovery and pairing,according to some embodiments. In some embodiments, the HMD 320 mustlearn of the existence of the TV device 310, and connect to it. In oneembodiment, the device discovery may use simple service discoveryprotocol, but discovery may also be achieved through mDNS and otherservices. The HMD 320 must then connect to the TV device 310 and networkwith it. This is called device pairing. Pairing may be accomplishedthrough WiFi. Bidirectional communication 305 is possible between theHMD 320 and the TV device 310. The HMD 320 then signals to the TV device310 that it is attempting to discover the position of the TV device 310in space. The TV device 310 then displays a pattern (e.g., a QR code, astatic or dynamic pattern, etc.) on its screen that the HMD 320 canrecognize through computer vision. The HMD recognizes the pattern andlearns the position of the TV device 310 in space. At this point it ispossible for the HMD 320 to create a holographic UI that is positionedrelative to the display of the TV device 310. The HMD 320 can utilizeits connection to the TV device 310 in order to send commands orretrieve information about the state of the TV device 310. For example,a holographic controller may be tightly integrated with the TV device310 UI (e.g., a Smart Hub UI in a Samsung Smart TV) to handlenavigation, focus, content selection, etc. The holographic controllerfollows a strict messaging protocol and handles messages between the TVdevice 310 and MR devices: The messages will provide variousfunctionality, such as remote app launching, remote game launching,volume control, video control (e.g., play, pause, stop, fast forward,and rewind), content selection from a TV device 310 UI, synchronizednavigation, channel control, etc. The TV device 310 also sends commandsto the HMD 320. As an example, the TV device 310 may be playing a movieon its screen. The TV device 310 reads metadata about the movie beingplayed that describes holograms to be inserted along with the moviecontent. Due to this metadata, the TV device 310 knows at what timecertain scenes or events occur in the movie. At the appropriate time,the TV device 310 signals to the HMD 320 to begin loading and cachingholographic assets. Finally, the TV device 310 may signal the HMD 320 toplay the holographic assets synchronized with the video on the TV device310.

In some embodiments, a holographic app discovers the TV device 310 on alocal network using SSDP protocol. The holographic app establishes aconnection to the TV device 310 and pairs with it. The TV device 310recognizes a new holographic app connection and opens a pairing app. Thepairing app shows an image on the TV device 310. The holographic apprecognizes the image and determines the location of the TV device 310 in3D space. Using an ARToolkit library (a software library for buildingaugmented reality applications), the holographic app uses image/patternrecognition to scan the image that the TV device 310 pairing appdisplays on the screen. Once the pattern is scanned, the holographic appbecomes aware of the location of the TV device 310 in the user's room.Then, the holographic app saves the location of the TV device 310 insideof its anchor store and uses this information each time the holographicapp is launched.

FIG. 4 shows an MR framework pairing flow diagram 400, according to someembodiments. The flow diagram 400 describes the device discovery andpairing process between the TV device (e.g., TV device 310, FIG. 3) andthe HMD (e.g., HMD 320, FIG. 3). Both a holographic app helper (e.g.,HoloEden Helper 410) and SEAL server 420, and the pairing App (e.g.,pairing app 430) are software on the TV device and the holographicclient (e.g., HoloTVClient library 440, and the holographic MR app(e.g., HoloEden MR app 450) and the SEAL Client 460 are software runningon HMD device.

FIG. 5 shows an example holograph app UI 520, according to someembodiments. The example UI menu holographs created by the HoloLens app.HoloLens app communicates with the TV device 310 via network to accessTV metadata information using an application programming interface (API)(e.g., an Eden API). For example, the holographic app obtains a listingof apps installed on the TV device 310, movie assets, title, etc. Acombination of the results of AR tracking and the display sendinginformation about itself (for example, its screen size) allow hologramsto be generated for holographic display around the TV device 310.Holograms are positioned next to the TV device 310, producing an effectof the UI “surrounding” the display. Note that the UI shows apps 520 andcontent (e.g., recommended media 510) that are available on the display.Interacting with the holographic tiles can trigger playback of contenton the display. There are also holograms for settings and controls 530including a virtual remote control, voice control, and other iconspositioned on the table in the room, made possible through utilizingroom-sensing technology on the holographic projection device (e.g., HMDdevice 320, FIG. 3).

In some embodiments, recommended media 510 is shown as a top panel, andprovides suggestions based on viewer trends. Holographic apps 520include user apps and games that appear in the menu below the screen.The selected app will appear to whichever side of the TV device 310 thatbest suits the user. Advertising (e.g., Tile Ads) opportunities appearto the left. Settings and controls 530 include settings, source, voicecommand and virtual remote are accessed from these panels.

FIG. 6 shows a virtual TV UI panel, according to some embodiments. Theholographic navigate and launch app 610 navigates via gazing, andtapping a selection (e.g., using a finger gesture) will bring up theapplication. The holographic pagination 620 provides for users toadvance through the media library. The virtual tabs 630 holographicprovides for users to switch between TV Shows, movies and recommendedmedia 510 by gazing at these tabs until a timer icon animates. Theholographic for media selection 640 provides for navigation via gazing,and tapping a selection brings up the preview panel.

FIG. 7 shows a virtual preview panel 710, according to some embodiments.The preview panel 710 begins automatically. The holographic for calls toaction 720 provides for users to tap to either expand the informationsection, make a purchase or close the preview panel.

FIG. 8 shows an example movie event holographic display, according tosome embodiments. The holographic virtual event 810 provides that duringkey moments in a movie, virtual events may occur outside the screen ofthe TV device 310 and even be interacted with. Here, the user may usethe HMD device 310 (FIG. 3) reticule to fire upon, for example,spaceships. In this example, during media mode, the holographic panels(e.g., pagination 620) are hidden for the user (hidden panels 820).Closing the movie will reveal the panels that were previously displayed.The holographic for the media player 830 show as a transparent media barand displays media time and options. In some embodiments, the mediaplayer 830 becomes fully opaque once the user gazes at it.

FIG. 9 shows a block diagram 900 for holographic object insertionprocessing, according to some embodiments. TV app processing 910 isshown on the left and the MR app 920 processing is shown on the right.In some embodiments, the TV app 910 reads content video metadataincluding a URL to a holographic description JSON. The MR device (e.g.,HMD 320, FIG. 3) is instructed to load a holographic description URL.The MR app 920 then parses the holographic description metafile JSONinto the schedule and loads asset bundles. The MR app 920 beginspreloading and scenes that are expected to play within a few seconds ofthe content starting. The MR app 920 signals to the TV device (e.g., TVdevice 310, FIG. 3) that preloading is ready. The TV app 910 beginsplayback of the content and continuously sends play head data to the MRdevice. The MR app 920 plays holographic scenes according to themetafile JSON.

FIG. 10 shows an example of dynamic advertising insertion in a virtualworld environment, according to some embodiments. During key moments ina commercial, virtual events may occur outside the screen and even beinteracted with. The holographic virtual event 1010 shows a RED BULL®airplane flying around the room environment outside of the display ofthe TV device 310. Products displayed in commercials and movies may beplaced in the user's environment and shown as holographic virtualproduct placement 1020. In this example, the virtual product placement1020 includes a can of RED BULL® placed on a table in front of the TVdevice 310.

FIG. 11 shows an example of a smart TV (TV device 310) and UI, accordingto some embodiments. In this example, the holographic recommended media510, control and settings 530, apps 520 and content selection 1110 areshown (via the HMD 320, FIG. 3). As can be seen, the collection of theholographic displays frees the TV device 310 from showing these UIs andprovides for varied placement and sizing for the different UIs.

FIG. 12 shows a block diagram of an example system layout, according tosome embodiments. The system includes a media device 1210 (e.g., a TVdevice, screen device, monitor, etc.), an HMD device 1220 (e.g., similarto HMD device 320, FIG. 3), an external server 1230, a social platformsynchronization server 1240 and a 3D asset store 1250.

In some embodiments, the media device 1210 may include a media devicecontrol server, an ad framework, HMD communications (comms), a pairingapp and a content playback app. The media device control server providescontrol of some media device functions. The ad framework may be apreexisting dynamic ad insertion framework, which is extended to handlereading metadata files for loading holographic content. The HMD comms isa communication protocol peer client that sends and receives messagesbetween the HMD device 1220 and the media device 1210. The pairing appreads information about the media device 1210 and presents a marker forthe HMD device 1220 to pinpoint a physical location of the media device1210. The content playback app is a custom video playback applicationthat reads metadata files for synchronization of 3D content, andincorporates the ad framework for synchronized ad playback. The mediadevice 1210 requests load/unload of a 3D asset from/to the HMD device1220, requests video ad serving template (VAST) and video multiple adplaylist (VMAP) from the external server 1230, and requests content fromthe external server 1230.

In some embodiments, the HMD device 1220 may include a media devicecontrol client, an HMD media device UI and HMD comms. In one embodiment,the media device control client is an HMD device 1220 side media devicecommunication client that listens for messages from the media device1210. The HMD media device UI is a 3D graphical user interface (GUI)that extends the media device 1210 UI into the user's physical space.The user physically interacts with the HMD media device UI when usingthe HMD device 1220. The HMD device 1220 finds the TV (media device1210), sends commands to the TV and requests information (e.g.,metadata) from the TV. The HMD device 1220 requests a 3D asset from theexternal server 1230.

In some embodiments, the external server 1230 may include an ad serverand a content server. The ad server that includes metadata for a 3D adwhen serving ads to the media device 1210. The content server is aserver hosting video/audio content.

In some embodiments, the 3D asset store 1250 may be a business ecosystemallowing content creators and publishers to promote and share 3Dcontent. The 3D asset store may be consumed by users, advertisers,publishers.

In some embodiments, the social platform synchronization server 1240handles synchronization of user group activities.

In some embodiments, holograms may not be projected from the HMD device1220, but instead there may be a separate device in the room, on theuser's person, or even integrated into the screen itself. The contentsynchronization aspect and learning of the display's position in spaceis similar as with the HMD device 1220. The screen and holographicprojection device may synchronize with each other over the Internet,BLUETOOTH®, or other communication technology, instead of over WiFi.

FIG. 13 shows an example diagram showing the sharing of a viewingexperience by synchronizing media devices and projecting avatars inrespective space, according to some embodiments. In this example, thereare three TV devices 1310, 1320 and 1330, which each have respectiveviewers that have virtual avatars (avatars 1340, 1341 and 1342). With asocial TV platform using the social platform synchronization server 1240(FIG. 12), users are able to watch video content on their respective TVdevice and communicate with each other, without being in the same realphysical space. Each user joins a “virtual room” to view the same videocontent that is synchronized for the TV devices 1310, 1320 and 1330. Therespective users in a given “virtual room” control the video content.Users in a “virtual room” are able to uniquely identify each other bythe design of their virtual avatar. Each respective user wears arespective HMD device (e.g., HMD device 320, FIG. 3, HMD device 1220,FIG. 12) that includes sensors (e.g., accelerometer, gyroscope, compass,etc.) that track head movements and position of the wearer. The virtualavatars (1340, 1341 and 1342) portray the head movement of the userwhere the avatar is facing a respective direction in which a user isgazing. The respective users in a “virtual room” are able to speak andhear the other users. Users in the social TV platform are able toadd/remove each other via a “friend list.” Virtual avatars may bepositioned relative to the display device (e.g., the TV, etc.) in thevirtual room.

FIG. 14 shows use of simultaneous visual input and network communicationfor providing precise determination of distance and position in 3Dspace, according to some embodiments. In one example, the TV device 1410includes content 1430, and the TYV device 1420 also includes content1430. In some embodiments, detecting the position of the display isperformed by using AR tracking. In some embodiments, determining thedisplays' position may be accomplished using AR tracking. The HMD (e.g.,HMD 320, FIG. 3, HMD device 1220, FIG. 12) recognizes the content 1430using computer vision, and determines its position in world space. Inaddition to determining the position of the display, the HMD alsoreceives information necessary to determine the display's dimensions.The HMD devices for each TV device 1410 and 1420 is able to determinethe size and shape of the holograms to display based on determination ofdisplay size, position and content position.

FIG. 15 shows a block diagram of a process 1500 for coordinating an MRconfigured HMD with a separate media device to enable a synchronizeduser experience, according to some embodiments. In one embodiment, inblock 1510 the process 1500 includes establishing a communicationchannel (e.g., a wireless channel) between the HMD (e.g., HMD 320, FIG.3, HMD 1220, FIG. 12) and the media device (e.g., TV device 310, FIG. 3,media device 1210, FIG. 12). In block 1520 the process 1500 includesperforming, via the communication channel, accessing content on themedia device or executing control commands on the media device based onan interface (e.g., a holographic UI) displayed by the HMD. In block1530, the process 1500 may perform, via the communication channel,detection of media (e.g., media/video/audio content), presented by themedia device and synchronization of the display of content (e.g.,holographic content) associated with the detected media on the HMD.

In some embodiments, for process 1500 the media device is a displaydevice or an audio device. The HMD displays a holographic UI associatedwith the media device. The holographic UI includes selectable UIelements (e.g., tiles, pages, etc.) for at least one of: accessingapplications (e.g., a UI of apps for selection/execution) on the mediadevice, accessing other content on the media device (e.g., movies,suggested content, etc.), controlling settings of the media device(e.g., remote control type selections) or modifying settings on themedia device (e.g., TV settings, such as source, TV settings,microphone, etc.).

In some embodiments, process 1500 may further include dynamicallygenerating a holographic asset (e.g., a holographic flying airplaneshown in the room environment) based on and associated with the mediacontent (e.g., an advertisement, commercial, movie, etc.) presented bythe media device. Process 1500 may also include that the media contentpresented by the media device is synchronized with display of theholographic asset on the HMD. In one embodiment, the MR content on theHMD includes the holographic asset.

In some embodiments, for process 1500 the holographic asset is retrievedfrom a holographic asset store (e.g., 3D asset store 1250, FIG. 12) thatmaintains holographic assets. The holographic assets may be sold andloaded to an HMD while the HMD views content on the media device.

In some embodiments, for process 1500 a timing for loading theholographic asset while the HMD views content on the media device isspecified by metadata provided by a third party associated with theholographic asset. The media device instructs the HMD to load theholographic asset based on the timing specified by the metadata.

In some embodiments, for process 1500 the holographic asset may be aninteractive advertisement that is dynamically inserted via the HMD upona triggered event by the media device. The holographic asset isretrieved from a holographic asset store and selected based on userprofiling data.

In some embodiments, for process 1500 the holographic asset triggered bythe media device is used to enhance e-commerce shopping experiences byproviding interaction with the holographic asset. Upon detection of theinteraction with the holographic asset via the HMD, an option ispresented to execute an action to facilitate a transaction associatedwith the holographic asset.

In some embodiments, process 1500 may further include providing one ormore holographic avatars using the HMD for other users engaged in anactive communication session with a user of the HMD. At least some ofthe users are at physical remote locations (i.e., at different physicallocations) and participate in the active communication session withrespective HMDs and media devices. The holographic avatars are spatiallypositioned relative to a particular media device, and media content issynchronized for presentation by each media device during the activecommunication session.

In some embodiments, process 1500 may further include spatially aligningUI elements displayed by the HMD with the media device based oninformation for one or more of: position, orientation, or size of themedia device. Spatial alignment includes: displaying, on the mediadevice, an image that provides information about the position or theorientation of the media device, or receiving, from the media device andvia the communication channel, size information for the media device.

FIG. 16 is a high-level block diagram showing an information processingsystem comprising a computer system 1600 useful for implementing thedisclosed embodiments. Computer system 1600 may be incorporated in adevice 52, 56, FIG. 1, or devices 210 or 250, FIG. 2, media device 1210,HMD device 1220, FIG. 12, etc.). The computer system 1600 includes oneor more processors 1601 (e.g., processor(s) 700, FIG. 2), and canfurther include an electronic display device 1602 (for displaying video,graphics, text, and other data), a main memory 1603 (e.g., random accessmemory (RAM)), storage device 1604 (e.g., hard disk drive), removablestorage device 1605 (e.g., removable storage drive, removable memorymodule, a magnetic tape drive, optical disk drive, computer readablemedium having stored therein computer software and/or data), userinterface device 1606 (e.g., keyboard, touch screen, keypad, pointingdevice), and a communication interface 1607 (e.g., modem, a networkinterface (such as an Ethernet card), a communications port, or a PCMCIAslot and card). The communication interface 1607 allows software anddata to be transferred between the computer system and external devices(e.g., over communication path 54, FIG. 1). The system 1600 furtherincludes a communications infrastructure 1608 (e.g., a communicationsbus, cross-over bar, or network) to which the aforementioneddevices/modules 1601 through 1607 are connected.

Information transferred via communications interface 1607 may be in theform of signals such as electronic, electromagnetic, optical, or othersignals capable of being received by communications interface 1607, viaa communication link that carries signals and may be implemented usingwire or cable, fiber optics, a phone line, a cellular phone link, anradio frequency (RF) link, and/or other communication channels. Computerprogram instructions representing the block diagram and/or flowchartsherein may be loaded onto a computer, programmable data processingapparatus, or processing devices to cause a series of operationsperformed thereon to produce a computer implemented process.

In some embodiments, processing instructions for 700 (FIG. 2) may bestored as program instructions on the memory 1603, storage device 1604and the removable storage device 1605 for execution by the processor1601.

Embodiments have been described with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products. Each block of such illustrations/diagrams, orcombinations thereof, can be implemented by computer programinstructions. The computer program instructions when provided to aprocessor produce a machine, such that the instructions, which executevia the processor create means for implementing the functions/operationsspecified in the flowchart and/or block diagram. Each block in theflowchart/block diagrams may represent a hardware and/or software moduleor logic. In alternative implementations, the functions noted in theblocks may occur out of the order noted in the figures, concurrently,etc.

The terms “computer program medium,” “computer usable medium,” “computerreadable medium”, and “computer program product,” are used to generallyrefer to media such as main memory, secondary memory, removable storagedrive, a hard disk installed in hard disk drive, and signals. Thesecomputer program products are means for providing software to thecomputer system. The computer readable medium allows the computer systemto read data, instructions, messages or message packets, and othercomputer readable information from the computer readable medium. Thecomputer readable medium, for example, may include non-volatile memory,such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM,and other permanent storage. It is useful, for example, for transportinginformation, such as data and computer instructions, between computersystems. Computer program instructions may be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

As will be appreciated by one skilled in the art, aspects of theembodiments may be embodied as a system, method or computer programproduct. Accordingly, aspects of the embodiments may take the form of anentirely hardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,aspects of the embodiments may take the form of a computer programproduct embodied in one or more computer readable medium(s) havingcomputer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readablestorage medium. A computer readable storage medium may be, for example,but not limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples (a non-exhaustivelist) of the computer readable storage medium would include thefollowing: an electrical connection having one or more wires, a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), an optical fiber, a portable compact disc read-onlymemory (CD-ROM), an optical storage device, a magnetic storage device,or any suitable combination of the foregoing. In the context of thisdocument, a computer readable storage medium may be any tangible mediumthat can contain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Computer program code for carrying out operations for aspects of one ormore embodiments may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of one or more embodiments are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products. It will be understood that eachblock of the flowchart illustrations and/or block diagrams, andcombinations of blocks in the flowchart illustrations and/or blockdiagrams, can be implemented by computer program instructions. Thesecomputer program instructions may be provided to a special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, createmeans for implementing the functions/acts specified in the flowchartand/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments. In this regard, each block in the flowchart or blockdiagrams may represent a module, segment, or portion of instructions,which comprises one or more executable instructions for implementing thespecified logical function(s). In some alternative implementations, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. It will also be noted that each block of the block diagramsand/or flowchart illustration, and combinations of blocks in the blockdiagrams and/or flowchart illustration, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts or carry out combinations of special purpose hardware and computerinstructions.

References in the claims to an element in the singular is not intendedto mean “one and only” unless explicitly so stated, but rather “one ormore.” All structural and functional equivalents to the elements of theabove-described exemplary embodiment that are currently known or latercome to be known to those of ordinary skill in the art are intended tobe encompassed by the present claims. No claim element herein is to beconstrued under the provisions of 35 U.S.C. section 112, sixthparagraph, unless the element is expressly recited using the phrase“means for” or “step for.”

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the embodiments has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the embodiments in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention.

Though the embodiments have been described with reference to certainversions thereof; however, other versions are possible. Therefore, thespirit and scope of the appended claims should not be limited to thedescription of the preferred versions contained herein.

What is claimed is:
 1. A method of coordinating a mixed-reality (MR)configured head-mounted display (HMD) with a separate media device toenable a synchronized user experience, the method comprising:establishing a communication channel between the HMD and the mediadevice; and performing, via the communication channel, at least one of:accessing content on the media device or executing control commands onthe media device based on an interface displayed by the HMD; ordetecting media content presented by the media device and synchronizingdisplay of MR content on the HMD and the detected media content.
 2. Themethod of claim 1, wherein: the media device is a display device or anaudio device; and the HMD displays a holographic user interface (UI)associated with the media device, the holographic UI includes selectableUI elements for at least one of: accessing applications on the mediadevice, accessing other content on the media device, controllingsettings of the media device or modifying settings on the media device.3. The method of claim 1, further comprising: dynamically generating aholographic asset based on and associated with the media contentpresented by the media device; and synchronizing the media contentpresented by the media device with display of the holographic asset onthe HMD, wherein the MR content on the HMD includes the holographicasset.
 4. The method of claim 3, wherein the holographic asset isretrieved from a holographic asset store that maintains holographicassets, and the holographic assets are sold and loaded to the HMD whilethe HMD views content on the media device.
 5. The method of claim 3,wherein a timing for loading the holographic asset while the HMD viewscontent on the media device is specified by metadata provided by a thirdparty associated with the holographic asset, and the media deviceinstructs the HMD to load the holographic asset based on the timingspecified by the metadata.
 6. The method of claim 3, wherein theholographic asset is an interactive advertisement that is dynamicallyinserted via the HMD upon a triggered event by the media device, and theholographic asset is retrieved from a holographic asset store andselected based on user profiling data.
 7. The method of claim 6, whereinthe holographic asset triggered by the media device is used to enhancee-commerce shopping experiences by providing interaction with theholographic asset, and upon detection of the interaction with theholographic asset via the HMD, an option is presented to execute anaction to facilitate a transaction associated with the holographicasset.
 8. The method of claim 1, further comprising providing one ormore holographic avatars using the HMD for other users engaged in anactive communication session with a user of the HMD, wherein at leastsome of the users are at physical remote locations and participate inthe active communication session with respective HMDs and media devices,the holographic avatars are spatially positioned relative to aparticular media device, and media content is synchronized forpresentation by each media device during the active communicationsession.
 9. The method of claim 1, further comprising spatially aligningUI elements displayed by the HMD with the media device based oninformation for one or more of: position, orientation, or size of themedia device, wherein spatial alignment includes: displaying, on themedia device, an image that provides information about the position orthe orientation of the media device; or receiving, from the media deviceand via the communication channel, size information for the mediadevice.
 10. An apparatus comprising: a memory storing instructions; andat least one processor executes the instructions including a processconfigured to: establish a communication channel between a mixed-reality(MR) head-mounted display (HMD) and a media device; and perform usingthe communication channel, at least one of: accessing content on themedia device or executing control commands on the media device based onan interface displayed by the HMD; or detecting media content presentedby the media device and synchronizing display of MR content on the HMDand the detected media content.
 11. The apparatus of claim 10, wherein:the media device is a display device or an audio device; and the HMDdisplays a holographic user interface (UI) associated with the mediadevice, the holographic UI includes selectable UI elements for at leastone of: accessing applications on the media device, accessing othercontent on the media device, controlling settings of the media device ormodifying settings on the media device.
 12. The apparatus of claim 10,wherein: the process is further configured to: dynamically generate aholographic asset based on and associated with the media contentpresented by the media device; and synchronize the media contentpresented by the media device with display of the holographic asset onthe HMD, wherein the MR content on the HMD includes the holographicasset; the holographic asset is retrieved from a holographic asset storethat maintains holographic assets; the holographic assets are sold andloaded to an HMD while the HMD views content on the media device; atiming for loading the holographic asset while the HMD views content onthe media device is specified by metadata provided by a third partyassociated with the holographic asset; and the media device instructsthe HMD to load the holographic asset based on the timing specified bythe metadata.
 13. The apparatus of claim 12, wherein: the holographicasset is an interactive advertisement that is dynamically inserted viathe HMD upon a triggered event by the media device; the holographicasset is selected based on user profiling data; the holographic assettriggered by the media device is used to enhance e-commerce shoppingexperiences by providing interaction with the holographic asset; andupon detection of the interaction with the holographic asset via theHMD, an option is presented to execute an action to facilitate atransaction associated with the holographic asset.
 14. The apparatus ofclaim 10, wherein: the process is further configured to provide one ormore holographic avatars using the HMD for other users engaged in anactive communication session with a user of the HMD; at least some ofthe users are at physical remote locations and participate in the activecommunication session with respective HMDs and media devices; theholographic avatars are spatially positioned relative to a particularmedia device; and media content is synchronized for presentation by eachmedia device during the active communication session.
 15. The apparatusof claim 10, wherein: the process is further configured to: spatiallyalign UI elements displayed by the HMD with the media device based oninformation for one or more of: position, orientation, or size of themedia device; and spatial alignment includes: displaying, on the mediadevice, an image that provides information about the position or theorientation of the media device; or receiving, from the media device andvia the communication channel, size information for the media device.16. A non-transitory processor-readable medium that includes a programthat when executed by a processor performs a method comprising:establishing a communication channel between a mixed-reality (MR)head-mounted display (HMD) and a media device; and performing using thecommunication channel, at least one of: accessing content on the mediadevice or executing control commands on the media device based on aninterface displayed by the HMD; or detecting media content presented bythe media device and synchronizing display of MR content on the HMD andthe detected media content.
 17. The non-transitory processor-readablemedium of claim 16, wherein: the media device is a display device or anaudio device; the HMD displays a holographic user interface (UI)associated with the media device, the holographic UI includes selectableUI elements for at least one of: accessing applications on the mediadevice, accessing other content on the media device, controllingsettings of the media device or modifying settings on the media device;and the method further comprises: dynamically generating a holographicasset based on and associated with the media content presented by themedia device; and synchronizing the media content presented by the mediadevice with display of the holographic asset on the HMD, wherein the MRcontent on the HMD includes the holographic asset.
 18. Thenon-transitory processor-readable medium of claim 17, wherein: theholographic asset is retrieved from a holographic asset store thatmaintains holographic assets; the holographic assets are sold and loadedto an HMD while the HMD views content on the media device; a timing forloading the holographic asset while the HMD views content on the mediadevice is specified by metadata provided by a third party associatedwith the holographic asset; the media device instructs the HMD to loadthe holographic asset based on the timing specified by the metadata; theholographic asset is an interactive advertisement that is dynamicallyinserted via the HMD upon a triggered event by the media device; theholographic asset is selected based on user profiling data; theholographic asset triggered by the media device is used to enhancee-commerce shopping experiences by providing interaction with theholographic asset; and upon detection of the interaction with theholographic asset via the HMD, an option is presented to execute anaction to facilitate a transaction associated with the holographicasset.
 19. The non-transitory processor-readable medium of claim 17,wherein: the method further comprises providing one or more holographicavatars using the HMD for other users engaged in an active communicationsession with a user of the HMD; at least some of the users are atphysical remote locations and participate in the active communicationsession with respective HMDs and media devices; the holographic avatarsare spatially positioned relative to a particular media device; andmedia content is synchronized for presentation by each media deviceduring the active communication session.
 20. The non-transitoryprocessor-readable medium of claim 16, wherein the method furthercomprises: spatially aligning UI elements displayed by the HMD with themedia device based on information for one or more of: position,orientation, or size of the media device; and spatial alignmentincludes: displaying, on the media device, an image that providesinformation about the position or the orientation of the media device;or receiving, from the media device and via the communication channel,size information for the media device.